-
Notifications
You must be signed in to change notification settings - Fork 324
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding indices on jobs table #2161
Conversation
Codecov Report
@@ Coverage Diff @@
## main #2161 +/- ##
============================================
- Coverage 76.60% 75.78% -0.82%
+ Complexity 1136 1061 -75
============================================
Files 219 209 -10
Lines 5287 5006 -281
Branches 420 403 -17
============================================
- Hits 4050 3794 -256
+ Misses 765 763 -2
+ Partials 472 449 -23
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Having index on reference columns seems to be a good idea most of the time 🥇 🚀 💯 |
@phixMe do we want to have this in the next release? |
@mobuchowski Yeah, let's get this in. |
Problem
Our current
/lineage
query is a little slow in instances where there are many jobs. When looking at the query plan, lots of the time is spent in the jobs table with the join operation. We are joining on some unindexed fields via thejobs_view
query that merges thejobs
table and thejobs_fqn
table for some semantic simplicity with our data model.Solution
Adds indices to the fields used we join on inside the lineage query.
Checklist
CHANGELOG.md
with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary).sql
database schema migration according to Flyway's naming convention (if relevant)